Spatial orientations of visual word pairs to improve Bag-of-Visual-Words model

This paper presents a novel approach to incorporate spatial information in the bag-of-visual-words (BoVW) model [1, 3] for category level and scene classification. In the traditional BoVW model, feature vectors are histograms of visual words. This representation is appearance based and does not contain any information regarding the arrangement of the visual words in the 2D image space. In this framework, we present a simple and efficient way to infuse spatial information. Particularly, we are interested in explicit global relationships among the spatial positions of visual words. For that we first introduce the notion of Pair of Identical visual Words (PIW) defined as the set of all the pairs of visual words of the same type. Then a spatial distribution of words is represented as a histogram of orientations of the segments formed by PIW. Figure 1 shows an example which gives an intuition to better understand our approach. Our method eliminates a number of drawbacks from the previous approaches [2, 3] by i) proposing a simpler word selection technique that supports fast exhaustive spatial information extraction, ii) enabling infusion of global spatial information, iii) being robust to geometric transformations like translation and scaling. In the conventional BoVW model, each image is represented by a set of local descriptors {d1 . . .dn} extracted from n patches around interest points or regular grids. A visual vocabulary W = {w1,w2,w3,w4 . . .wN} is obtained by clustering a set of descriptors from all the training images. Here, N is a predefined number and the size of the vocabulary. Each patch of the image is then mapped to the nearest visual word according to the following equation: